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ABSTRACT 

This is the report of a study designed to test a 
model for determining the effectiveness of sets of instructional 
materials which can be considered hierarchical and to derive a 
mathematical relationship by which such measures can be quantified. 
The model tested requires that the instructional materials be 
hierarchical in character. The Introductory Physical Science (IPS) 
course was used in the study. The public schools in a suburban county 
provided the instructional setting. The model for measuring 
effectiveness proved, within the constraints of the study, to be a 
valid one when applied to the performance of boys in the IPS course. 
It worked especially well for boys of high ability. The model was not 
applicable in the case of girls* performance in the IPS course. The 
report also contains the investigator's evaluation of the IPS course 
and materials, some of the evaluation instruments used, and a 
bibliography. (LC) 
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PREFACE 



The author owes a large debt of gratitude to the administration and 
to the professional staff of The Fairfax County public schools. Without 
their cooperation and their dedication to Improved Instruction In their 
county's schools this study could not have been conducted. The results 
of this study should not be taken to mean that any school In the county 
was not doing a good job of teaching IPS. The models used to measure 
the effectiveness of the IPS course are empirical and, were they perfected, 
would provide only a small input to any evaluation of the effectiveness of 
a school's science course. 



TABLE OF CONTENTS 



Page 



PREFACE • 1 

LIST OF TABLES 3 

LIST OF FIGURES ^ 

SUMMARY 5 

I. BACKGROUND OF THE STUDY b 

Introduction 

Review of Literature 9 

Theoretical Basis for Models on Structure . . 13 

and Effectiveness 17 

II. THE STUDY 17 

Problem 17 

Method 19 

III. RESULTS 23 

Incomplete Data . 23 

Identification of Relevant Basic Abilities. . 25 

Measures of Effectiveness 51 

IV. CONCLUSIONS AND RECOMMENDATIONS 61 

Conclusions 

Recommendations 63 

V. REFERENCES 65 

VI. APPENDICES b7 

A - The IPS Course 68 

B - Development of an IPS Student Checklist • 87 

C - IPS Student Checklist-Preliminary 

Version . « . 9b 

D - IPS Student Checklist - Revised Form. . . 102 

E - Characteristics of Participating Schools. 107 

F - Form For Collecting Data 117 

G - Regression ModeLls Used in Completing . . 

Partial Records 119 

VK. BIBLIOGRAPHY 122 



- 2 - 



o 



I ERIC 



LIST OF TABLES 



Table 



Page 



1. Correlation Coefficients for Scores on Abilities Not 

Relevant to Hie IPS Course with Scores on Achievement 

For High Ability Students 27 

2. Correlation Coefficients for Scores on Abilities Not 

Relevant to The IPS Course with Scores on Achievement 

For Low Ability Students 30 

3. Coefficients of Correlation of IQ with Achievement for 

Hig£ and Low Ability Girls 37 

4. Coefficients of Correlation of IQ with Achievement for 

High and Low Ability Boys * . 40 

5. Coefficients of Correlation of Reading Ability with 

Achievement for Low Ability Girls 42 

6. Coefficients of Correlation of Space Relations Ability 

With Achievement for High and Low Ability Boys. .... 45 

7. Coefficients of Correlation of Verbal Reasoning Ability 

with Achievement for High and Low Ability Boys 48 

8. Summary of The Test of Predicted Behaviors of 

Measured Abilities 50 

9. Summary of Comparison of Predicted Changes in 

Achievement with Actual Changes for Higfc Ability 

Boys using Verbal Reasoning Ability as Relevant 

Basic 53 

10. Summary of Comparison of Predicted Changes in Achievement 

with Actual Changes for liagb Ability Boys Using Verbal 
Reasoning as Relevant Basic 5 0 

11. Coefficients of Correlation of Numerical Ability with 

Achievement for High and Low Ability Boys 59 

12. Summary of Measures of Hie Effectiveness of The IPS 

Course for High and Low Ability Boys 60 




- 3 - 



\J1 



LIST OF FIGURES 



Figure 



Page 



1. Pattern £ of Coefficients of Correlation of IQ with 

Achieve. in, r it for High Ability Girls 

2. Patterns of Coefficients of Correlation of IQ with 

Achievement for High Ability Boys 

3. Pr i^erjjs. of Coefficients of Correlation of IQ with 

Achievement for Low Ability Boys 



4. Patterns of Coefficients of Correlation of Reading 

Ability with Achievement for Low Ability Girls • . . 

. Patterns of Coefficients of Correlation of Space 
Relations Ability with Achievement of High 
Ability Boys 

6 . Patterns of Coefficients of Correlation of Space 

Relations Ability with Achievement for Low 
Ability Boys 

7. Patterns of Coefficients of Correlation of Verbal 

Reasoning Ability with Achievement for High 
Ability Boys 

8 . Patterns of Coefficients of Correlation of Verbal 

Reasoning Ability with Achievement for Low 
Ability Boys 

9 . Patterns of Coefficients of Correlation of Numerical 

Ability with Achievement for High Ability Boys . . . 

10. Patterns of Coefficients of Correlation of Numerical 
Ability with Achievement for Low Ability Boys . . . 



41 



43 

44 



47 



49 

57 

5» 




- 4 - 



SD4MAKY 



The difficulty of obtaining sound objective measures of the 
effectiveness of sequences of instructional materials is encountered * 

by every evaluator who needs such measures as an aid in selecting ^ 

materials for assignment to schools, to teachers, and to students. 

There has been enough research done on the effectiveness of sequences * 

of instructional materials which are hierarchical in character to 
make it feasable to attempt a quantification of measures of effect- 
iveness. This study represented one attempt to test or model for 
determining the effectiveness of sets of instructional materials 
which can be considered hierarchical and to derive a mathematical 
relationship by which such measures can be quantified. 

The model tested requires that the instructional materials 
be hierarchical in character, not true hierarchies. The Introductory 
Physical Science course satisfied this requirement and was the one used 
in the study. The public schools in a suburban county provided the 
instructional setting and assured a severe test of The Model. 

The model for determining effectiveness depended upon the 
untested assumption that the instructional materials used were in- 
deed hierarchical in character. A model to test this assumption 
was also available, and that model too was to be tested in The 
Study. The model, could be tested, however, only if at least some 
groups of IPS students completed the entire course. 

The model for measuring effectiveness proved, within the 
constraints imposed by the study, to be a valid one when applied to 
the performance of boys in the IPS course. It worked especially 
well for boys of high ability. The model was not applicable in the 
case of girls' performance in the IPS course. 

Because no class of student;, completed the entire IPS 
course, the model for testing the hierarchical character of the j 

course could not be employed. Neither could there be a quantification 
of the measures of effectiveness of the IPS courses for such 

quantification depended on students making more progress than they 4 

actually made . 

Further tests of the model should be done under carefully ", 

controlled situations. Although the model for determining effectiveness 
retained most of its promise, the requirement that it hold in a real 
school setting proved to severe* The model is so sinqple in concept > 

that it would be extremely valuable to educational evaluators if its 
use could be clearly justified by experiment. 
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BACKGROUND OF THE STUDY 



Introduction 



The problem of determining the effectiveness of sets of 
instructional materials in promoting student learning is among 
the most important problems facing the educational evaluator. It 
is at the same time one of the most difficult problems and is per- 
haps the problem handled most carelessly by evaluators. Evaluation 
of sets of instructional materials have at their best consisted of 
deciding the relative merits of two alternative sets of instructional 
objectives. At their worst they have consisted of conjecture, 
innuendo, propaganda, and in some cases outright falsehood designed 
primarily to sell an instructional product in the educational market- 
place. Methods which provide objective measures of the effectiveness 
of sets of instructional materials often end with the word counts and 
readability formulas, which are of limited value and which are approp- 
riate for only a small number of materials. Beyond this, few object- 
ive methods for measuring effectiveness exist. 



There are several reasons for the perplexing situation. The 
most important is that evaluation has been conceived in a very 
sense to include all the systematic efforts to assess the strengths 
and weaknesses of educational materials, (Grobman, 19°?) • The 
consequence of this concept being that evaluators have attacked their 
problems on too broad a front. While it is important to recognize 
that evaluation as a process in curriculum design is of broad scope, 
it is just as important to recognize that such a concept of evaluation 
is very seductive. It attracts to the field of evaluation many who 
are lured not by the satisfaction of performing useful evaluations 
and deriving productive methods but by the prestige to be had in be- 
coming associated with a respected and burgeoning field of study. 



As a consequence educational evaluators have not vigorously 
attacked the problems involved in adding to educational evaluation 
an element of empirical science. They have resisted the use of 
experimental methods in their search for factors influencing the 
effectiveness of instructional materials. They have not concerned 
themselves with such momentous problems as that of identifying the 
characteristics of instructional materials, characteristics of the 
learner, and teacher characteristics which go together to maximize 
leaning. They have not, as Gagne (Gagne, 1967) has suggested, been 
concerned with determining what dimensions of curriculum may be sys- . 
tematically varied to determine their efforts on the learning accomplished 
by the student. And they have not, I might add, always systematically 
varied such dimensions as are already known. 



Instead of attempting to devise and validate empirical 
methods of measuring such characteristics of instructional materials 
as their effectiveness, educational evaluators have too frequently 
taken the easier way, the descriptive approach to evaluation. 
Evaluations have as a result, often been highly opinionated descrip- 
tions which are more persuasive than convincing. Objective techniques 
for evaluation have become unnecessarily muddled because of too great 
a reliance on this approach. For whatever reasons; expediency, 
financial gain, personal gain, or whatever; the field of evaluation 
can no longer tolerate the consequences of taking the easy way out. 

If things are ever to become clearer, if the evaluator is ever to be 
able to identify and systematically vary the dimensions of curriculum 
which will yeild objective techniques for evaluation, then evaluators 
must face the task of devising empirical methods for accoraplibllng 
their work. 

There is abundant evidence to support the contention that 
evaluation in education is in a state of confusion and that empirical 
techniques, while desperately needed, are not ‘ .ng desperately sought. 
Since the advent of the highly flattered curriculum study groups and 
their programs with new goals and new priorities, large sums of money 
and vast quantities of time and energy have been invested in the 
development of complete courses for the secondary schools and the 
elementary schools of the United States. The products of several 
curriculum study groups have been used extensively in other countries. 
Evaluation of the products of these study groups has been mostly a 
matter of informal feedback during the process of developing the 
curriculum materials. This process has usually consisted of the 
following steps: 

1. Writing a preliminary set of materials. 

2. Preliminary trial of this set of materials. 

3. Revision of the set of materials using information 

from the preliminary trial. 

4 . Rewriting and second trial (repeated if necessary). 

5 . Production of a commercial set of materials. 

Upon the completion of a saleable sit of instructional 
materials, the products of the study group typically have been turned 
over to commercial publishing companies for distribution. While re- 
visions do not cease to be made in many of the materials, evaluation, 
such as ever existed, in most instances ceases. 

During the development of their products, the curriculum study 
groups did little to answer such relevant questions as; 

1. To what extent do the instructional materials promote 
the learning claimed for them? 



2. To what extent do the materials exhibit the structure 

(sequence) claimed for th3m? 

3. Do the materials indeed tee ch new knowledge in a general 

way, as is often claimed? 

Do the materials teach so that knowledge is more readily 
transferable? 

5. Are the new materials more effective than the old? 

These questions imply that underlying the development of the 
instructional materials produced by curriculum study groups were 
several untested assumptions basic and vital to successful evaluation 
of the materials. Furthermore, it may be inferred that these assump- 
tions involve variables which have not been identified and properly 
treated in experimental development of the products of the curriculum 
groups. If so, it follows that evaluations of such materials have been 
at best less than adequate. Unfortunately, the evaluation carried out 
by the various curriculum study groups represent the best, not the worst 
evaluations of instructional materials. That there are untested 
assumptions underlying the new curriculum materials is in fact the 
case. Many of the assumptions, along with suggestions of numerous 
problems in need of experimental study, can be found in the publication 
which resulted from the Woods Hole Conference, called in 
1959 by the Education Committee of the National Academy of Sciences, 
(Bruner, 1963 ) and attended by thirty-five scientists, scholars and 
educators. B.O. Smith (Ford and Pugno, 196*0 has given a concise 
statement c: some of the important assumptions underlying the develop- 
ment of many of the materials produced by curriculum study groups. 

These assumptions are (in part): (l) that "teaching will be more 

effective if it incorporates the ways elements of knowledge are re- 
lated logically (structure), (2) what is learned will be retained 
longer if it is tied into a meaningful structure, (3) what is learned 
will be more readily transferred if it is tied into a system of knowledge, 
and (*0 knowledge can be categorized in ways more conducive to learning..." 

William W. Stokes (Stokes, 196*0 studied the action taken on 
the recommendations of the Woods Hole Conference and concluded that 
the basic assumptions about the nature of knowledge and the nature of 
learning announced by the group that met there had not been carefully 
researched. He further concluded that the advantages claimed for 
teaching the structure of a discipline had not been researched and that 
research on the effectiveness of the curriculum projects based on dis- 
ciplinary structure was so ambiguous that it was not possible r co determine 
how effective the programs were. The situation has not changed much 
since Stokes* study, except that the impact of the curriculum studies 
on the public schools is now even more uncertain. 



While numerous examples of poorly conducted evaluations, the 
most flagrant examples probably being evaluations of Title HI (ESEA.) 
Programs, could be mentioned, this is unnecessary at this point. The 
unknown impact of the instructional materials produced by the curriculum 
study groups is alone justification for concern about developing more 
objective techniques for evaluation than are now available. The 
materials produced by these groups are widely used in the public 
schools of the United States and they have had a great deal of influence 
on materials and programs which were already in use. It would be 
difficult, if not impossible, to measure the full impact of the mat- 
erials produced by these groups! In many cases the materials them- 
selves have been used exclusively; in other instances they have 
provided inrputus for revisions and modifications in existing programs. 
There can be little doubt that the curriculum reform movement of the 
past decade has stimulated needed changes in school curricula. Given 
that the goals of the new curricula are new and that untested assump- 
tions underlie them, however, it is safe to say that evaluation of 
the materials is perhaps the weakest flank of the new movement; yet 
it is probably at the same time the most vital. The problem of testing 
and developing empirical methods for assisting in the mammoth task of 
evaluating instructional programs is the problem to which this study 
addressed itself. In the belief that the problems of evaluation will 
best be solved by attempting to develop sound empirical methods to 
support a more comprehensive concept of evaluation, the study focused 
on a narrow problem in the field of evaluation. This is in no way 
intended to detract from the work of those who choose to treat evalua- 
tion in its broader context. In fact, it is hoped that the atomistic 
approach taken in this study will help to clarify some of the problems 
in the broader field of evaluation and thereby make some contribution 
to that broader field. 



Review of Literature 

If one looks closely at the assumptions listed by Smith, 

(Ford and Pugno, ‘’.969) four dimensions of special concern for the 
evaluator of instructional sequences appear: effectiveness, structure, 

retention, and transfer. There was no attempt to deal in this study 
with all of them, but two of them were selected for study. They are 
closely related and, because I had previously dealt successfully with 
them in an earlier experiment, they lent themselves to further experi- 
mental study. The two dimensions are structure and effectiveness. 

What follows is an attempt to review research which has dealt 
with these two dimensions and to describe two models, developed and 
tested in my previous study, for measuring the effectiveness and the 
structure of instructional materials. The models used are appropriate 
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for naterials which can be said to nave structure, of which the best 
examples are in mathematics and science. While the models are not 
appropriate for all kinds of materials, they represent a great 
' improvement over existing models for evaluating instructional sequences. 



While research studies completed to date by no means answer all 
the Imp ortant questions about the relationships among structure, 
effectiveness, and the learner, several studies have been reported which 
provided the groundwork for desi gnin g and conducting the study reported 
here. Since they "i? important to understanding the models used in this 
study to measure effectiveness and structure, a brief review of the 
important studies will be given at this point. The review is followed 
by a brief discription of the models employed in the study being 
reported. 

j. Miller and S. Levine (1952) studied two different ways of 
using review sequences in films. They were concerned with the relative 
merits of (l) spacin g the review sequences throughout the film after 
each nftjor topic and (2) putting the entire review at the end of Hie 
film. The film used in the experiment was on Ohm's law. In the first 
review condition, each of four sections of the film was shown and 
immediately reviewed; in the second review condition, the entire film 
was shown and then reviewed. The study revealed that the complete 
showing and review at the end was the superior of the two conditions. 
The experimentors also considered the problem of the effect of the 
frequent use of subtitles on the effectiveness of the film. The sub- 
titles were used to identify and structure the several subtopxcs in the 
film on Ohm's Law. r ivo degrees of structuring were used, major sub- 
titles only and complete subtitling, and a control, in which the 
material flowed without a break, was employed. Even though the diff- 
erent review conditions had significantly different effects , no 
significant differences were found among the three structuring treat- 
ments. 



Wulff , and Stolurow (1967) did a study comparing two forms of 
organization for teaching aircraft rivet coding. The rivets were 
color coded for four properties: length, diameter, bead shape, and 

material. Two forms of or^'anization were used. One form presented 
the material so that differentiating cues for each item could be 
learned by classes; the other form presented the same information, 
but presented all the information about a given item at one time so 
that it wps unlikely that the learner could utilize cues learned by 
classes d u rii learning in this form. As the experimentors had pre- 
dicted, the method which utilized cues provided superior to the other 

method. 



Gavurin and Donahue (i960) conducted a study using programmed 
materials in psychology given in an ordered sequence to one group of 
adults and in a randan sequence to another group. They used a program 
with approximately ten-item blocks and required an errorless trial 
within a block before the subject could advance to the next block in 
the program. When an errorless trial was used as the criterion, the 
ordered sequence proved superior. A test on retention given one month 
later, however, revealed no significant differences between the group 
taking the ordered sequence and the group taking the random sequence. 

Levine and Baker (1963) believing error rate not to be a 
satisfactory criterion to use in evaluating a program on retention 
and transfer, conducted a study to determine the importance of pre- 
senting items in a standard, logical sequence. They used a program 
of units in geometry with second graders and treated one unit of this 
program experimentally by randomizing the frames in that unit. The 
experimenters found no significant differences in (l) median n umb er 
of errors, (2) mean working time, and (3) means on acquisition, reten- 
tion and transfer measures. They indicated that individual differences 
might have obscured treatment effects, and, that on the basis of test 
scores, the program failed to teach the material effectively. They 
did not advocate giving up the idea that sequence is an important 
variable, but suggested examining the size of the divisions used in 
scrambling the sequence. 

K. Roe, Case, and A. Roe (1962) examined the hypothesis that 
the mean performances for students who have studied a proper sequential 
ordering »nd students who have studied a random ordering^ the same 
items will be significantly different. The items used in their experi- 
ment were on elementary probability, were related, and each item 
normally depended upon the preceeding one. Only the student* s terminal 
performance was considered. The experimenters used a seventy-one-item 
program with two groups of eighteen psychology students each. One group 
was presented an ordered program and the other group a scrambled 
program. The experimenters found no significant differences in (l) time 
required for learning, (2) error score during learning, (3) criterion 
test score, and (h) time required for the criterion test. They suggested 
that sequence in an auto-instructional program may be a function of such 
variables as length of program, content of items, individual differences, 
and spacing of criterion measured. 

Bayne, Krathvohl, and Gordon (1967) investigated sequence in 
programmed instruction when the logical inter-relatedness of the material 
was varied. They hypothesized that the effect of scrambling would be 
greatest for the topics having the most internal logical development. 

The experiment involved 238 college sophomores in an elementary 
psychology course and eight combinations of linear and scrambled topics 
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differing in logical interrelatedness of the material. The experiment- 
ers found (l) no significant differences between the means of the 
eight groups on an immediate and a delayed test, (2) no significant 
effect due to degree of dependence of content of learning sequence, 
and (3) no significant relation between performance and ability. 

Gagne and Paradise (1961) have reported a study which gives 
much insight into the problem of testing a defined program for its 
sequence and for its effectiveness. These experimenters theorized 
that differences in the rate of acquisition of successive frames in 
a program depend upon the amount and kinds of knowlea that the 
learner brings to the learning task and not in as greai, a way on general 
intelligence. They conceived a hierarchy of learning sets at the bottom 
of which are very general sets and at the top of which is a final 
learning set. The general sets at the bottom of the hierarchy are 
called relevant basic abilities and are considered essential to the 
successful completion of the final learning set. Positive transfer 
is effected from set to set throughout the hierarchy, with attain- 
ment of the final set considered to be a matter of the successive 
attainment and assimilation of the sequence of lower sets, begi nnin g 
with the lowest learning set already available to the individual. 

According to hypotheses based upon this theoretical position, 
individual differences can be independently measured as differences in 
(l) general intelligence, (2) relevant basic abilities, and (3) number 
and pattern of relevant learning sets. Furthermore, an ideally effec- 
tive program, a program in which all lea rnin g sets are achieved by every 
subject, should reduce the variance attributable to number and pattern 
of re le vant learning sets to zero. If the program is not ideally 
effective, an increasing number of subjects will "drop out" as hi g h er 
levels of the hierarchy are reached. Those who "drop out” will tend 
to be subjects of low basic ability. The "dropping out" of subjects 
as higher levels are reached will be indicated by increasing correla- 
tions 'of relevant basic abilities with achievement at progressively 
hi gher levels in the hierarchy. These correlations and the rate at 
which they change can be used to measure the effectiveness of the program. 

On the basis of their experiment, the authors reported 
that correlations between basic ability and achievement confirmed 
prcdicitons based on the theory and that correlations between learning 
rate a nd relevant and irrelevant abilities also were in agreement with 
predictions based on theory. Transfer among learning sets was reported 
to be high, and a prediction that rate of learning depends decreasingly 
upon relevant basic abilities as learning progresses upwards in the 
hierarchy was confirmed. The authors indicated, however, 'that the 
i earn! ng program used in this study was only moderately successful. 
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Believing that the inconclusiveness of findings when studying 
the effects of sequence changes in instructional materials, whether 
they be programmed materials or not, was due to the failure to specify 
clearly what an ordered sequence of materials was to be and to test 
that sequence, X developed and tested models for measuring the 
structure (sequence) and the effectiveness of programmed instructional 
materials having a specified sequence, called structure. The models 
were based on a theoretical rationale, and since they are being used 
in the study being reported here, I elect to briefly describe them 
before reviewing the research study which reports their test. 

Theoretical Basis for Models On Structure and Effectiveness 

The results of reported research and the ideas upon which this 
research has been based provided the basis for a theoretical framework 
which served as a foundation for designing and conducting the study 
reported here. A brief discussion of this theoretical framework is 
given at this point, since the data analysis takes on its significance 
within that framework. 

In a program which is one-hundred-percent effective, all the 
students can be expected to achieve all the intended elements in the 
program. In a program that is less than one -hundred-percent effective, 
the students who will achieve least are these who score lowest on tests 
used to measure certain relevant basic abilities necessary for success- 
ful achievement of the total program. Therefore, correlations between 
the scores on tests which measure basic abilities and achievement at 
successive points in a hierarchical program provide a measure of the 
effectiveness of the program. A set of high but rapidly increasing 
correlations of initial relevant basic ability with achievement at 
successive points, indicating that students of low basic ability are 
not achieving at the higher points, will be an indication of an in- 
effective program. A set of high but rapidly decreasing correlations, 
indicating that most students are "over-achieving" at the higher points, 
will be an indication of an effective but an inefficient program. A 
set of high correlations of close to zero slope, when plotted against 
distance in the hierarchical program, will indicate an effective 
program. A slight positive slope would also be indicative of a 
structured (hierarchical ) unit . 

If a program is hierarchical, achievement at successive points 
is dependent upon the achievement up to and including each previous 
point. As a consequence, the scores on achievement tests to a point 
should be good predictors of achievement at the next point in the 
hierarchy. If a program were one -hundred-percent effective, that is, 
if everyone achieved everything intended in the program, it would be 
difficult to determine whether the program actually constituted a 
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hierarchy without readministering it omitting certain parts. No 
program, however, is ideally effective, and thus regression analysis 
can he used as a tool to examine the hierarchy of a program. The 
assumption underlying this as a technique for deciding the question 
of hierarchy is that, in a hierarchy, each point provides positive 
transfer to the next point in the hierarchy. The extent to which 
this positive transfer is acting within the program can be used to 
check to extent to which the program can be considered hierarchical. 

In a set of partial regression coefficients using achievement at 
several points in a hierarchy to predict final achievement of the 
hierarchy, each predictor should carry a significant weight with the 
weights show ing a tendency to decrease because of the increasing 
uncertainty of predicting the subsequent score of one who achieves at 
a given point. One who fails would almost certainly continue to fail 
in a true hierarchy. If this is indeed the case, correlations of final 
achievement scores with achievement scores upward through the hierarchy 
should exhibit the same decreasing pattern. If basic ability alone 
accounted for this decrease in correlation coefficients, correlations 
with basic ability held constant would not be expected to exhibit this 
decreasing pattern; in fact the pattern could conceivably be reversed. 

Basic ability would be expected to be most important in 
predicting achievement at the beginning of the hierarchy. Because 
achievement at later points in the hierarchy comes to depend more on 
achievement at preceding points and less on basic ability, the 
importance of basic ability as a predictor of final achievement should 
decrease as one goes upwards in the hierarchy (Gagne and Paradise, 19 ° 1 )« 
The intercorrelations of the successive tests in the hierarchy might be 
expected to increase because of the factor giving rise to the decreasing 
correlations of final achievement with achievement upwards through the 
hierarchy. 

Experimental Test of the Models 

study which provided a test for the models (Pyatte, 19^9) 
used a programmed uni t or measurement written to conform to a definition 
of structure. The criteria for a structured unit were derived from 
the criteria for structured courses in science, recently produced by 
the various curriculum groups, as follows: 

1. The course is developed around certain ideas or 

concepts which provide a logical and integrated 
picture of science and the science course. 

2. The course has a central theme which helps to hold 

the development of the concepts together. 
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3. The various parts ot the course (text, lab, etc.) 
are closely related. 

The course emphasizes knowledge and understanding. 

5. The course provides for the active involvement of 

the student in science-like problem situations. 

6. The course is developed so as to lead the student 

through an increasingly complex and elaborate 
understanding of the concepts included, toward 
an ultimate understanding of the desired structure 
of the course. (This type of development can be 
considered hierarchical in that more elaborate 
understandings depend upon less elaborate understandings.) 

7. The course provides periodic reviews of the concepts 

provid-d. 

The unit used in the test study was written as a hierarchy 
consisting of four steps. Achievement tests were administered at 
each of the four steps, and a transfer test was administered along 
with the fourth a ievement test. These two measures were used as 
criterion measures for another part of the study, which does not 
require explanation here. 

One version of the measurement unit was used as it was 
written. A second version consisted of the four steps — scrambled 
so that the order was one, four, three, two. Both versions of the 
unit were bound as a part of the larger course and were completed by 
172 students in fourth, fifth and sixth grades. The Arithmetic Skills 
test of the Iowa Itests of Basic Skills were used as a measure of basic 
arithmetical ability in the measurement of effectiveness of the two 
versions of the measurement unit. 

When the data were analyzed, the sets of correlation co- 
efficients revealed that the structured version of the measurement 
unit was effective and indicated that it was in fact structured as 
intended. The sets of correlation coefficients for the unstructured 
unit did not exhibit the expected pattern, so it was assumed that at 
least some of the structure had been lost in scrambling. 

The success of the two models developed and studies 
experimentally prompted the study which is being reported here. The 
intent of this study was to further test the model for measuring 
effectiveness and, providing it was successful, to attempt to arrive 
at a mathematical expression for determining the effectiveness of sets 
of instructional materials which could be considered structured. At 
the same time it was noped that the data gathered would shed some light, 
on the problem of measuring the extent of structure in such materials. 
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Since the measurement unit on which the models were originally 
tested was written to conform to a set of crieteria for a structured 
science course, it seemed logical to search for a science course which 
satisfied that set of criteria as nearly as possible for use in the 
study reported here. Furthermore, it was decided that an actual public 
school situation rather than a controlled experimental situation would 
provide a more rigorous test of the models. 

The course selected for study was the Introductory Physical 
Science Course. A detailed description of this course appears in 
Appendix A of this report, so no further description will be needed 
at this point. 



'The original tests were performed in the public schools but in a 
situation more carefully controlled than was the case in this study. 



THE STUDY 



Problem 



Is this set of instructional, materials effective? How 
effective is this set of instructional materials? Is this set of 
instructional materials more effective than that set? For which 
students is this set of instructional materials effective and for 
which is it not effective? 

Such questions about the effectiveness of sets of instructional 
materials are of very great importance to the evaluator as well as to 
the user of instructional materials designed for use in the classrooms 
of our schools, but they have not been given the attention by education- 
al researchers which questions of such importance demand. The major 
purpose of this study was to examine a prototype model for measuring 
tha effectiveness of a set of instructional materials which could be 
said to be structured and which was being used in a real school situation . 
It was hoped that a mathematical expression could be derived which could 
be used as a means of quantatively measuring the effecti veness of such 
sets of instructional materials. If successfully developed and tested, 
such a model would obviously be of great practical value to the evaluator 
of such instructional sequences. 

Is a structured set of instructional materials more effective 
than an unstructured set? Is this sequence of materials more effective 
than that one? These are questions which are closely related to the 
problems of measuring the effectiveness of sets of instructional 
materials and which will deserve much of the attention of future re- 
searchers in methods of evaluating instructional materials. A secondary 
objective of this study was the testing of a prototype model for measur- 
ing the extent to which a given set of instructional materials is~ 
structured . 

These are essential steps if comparisons are ever to be 
legitimately made among various instructional sequences and if the 
relationships among the variables relevant to learning styles, teaching 
styles, and modes of instruction are ever to become clear. 

The theoretical basis for the prototype models has already been 
explained, and no further attention will be devoted to it here. It 
might be said at this point, however, that the study was intended to 
test models already in a primitive state of development. The test was 
intended to require t! the models stand up under a real school situation. 
As will be pointed out, "when the results are discussed, this was perhaps 



an unrealistic demand. Nevertheless, lessons of considerable 
Importance were learned from the study, and, although the data were 
not as revealing as it had been hoped, the study was not without some 
success. The problems being considered in this study are of such 
great importance, and so little is known about them at this time, 
thc.t the results of this study deserve the careful attention of any- 
one who is involved in the evaluation of instructional programs 
designed for use in the schools. 



Method 



Subjects 



Seven schools in Fairfax County, Virginia, were selected to 
participate in the data collection for this study. One eighth grade 
teacher was selected from each of the seven schools. These teachers 
were chosen for their exceptional talent and experience with the IPS 
course so that the course would have a fair chance to do what it was 
designed to do. The teachers taught five classes each, giving a 
total of thirty-five classes involved in the study. The total number 
of students involved during the entire project in all the classes was 
'31. One hundred and forty students were excluded from this total 
^ecause they had incomplete data records which could not be completed. 2 
The records were incomplete because the students either transfered out 
of the system before completing the course, failed to complete the 
course, or missed large numbers of tests, Most of the l4o were transfer 
students . 

Materials 

The materials used in this study were: (l) The Introductory 

Physical Science Course with its accompanying materials and ( 2 ) The 
Differential Aptitude Tests, Form L. Some data were recorded using the 
cumulative records of the students. The data recorded from cumulative 
folders was intended to be used mainly to categorize students. The 
information considered important enough to be recorded was (l) age, 

( 2 ) sex, (3) IQ, and (4) reading ability (as measured by the ICWA 
Silent Reading Test, total score). In addition to information useful 
in categorizing students, IQ and reading scores were expected to be 
helpful in verifying expected relevant basic abilities. The DAT is 
widely used as an instrument for measuring aptitudes in one of several 
categories. It was used in this study to measure basic abilities 
relevant to the hierarchical science course as well as to provide some 
verification that abilities expected to be relevant basic behaved 
differently than those expected to be irrelevant or general abilities. 
The IPS course is described in detail in Appendix A, but a brief 
description will be given here. 

The Introductory Physical Science (IPS) course consists of a 
textbook, a Teacher’s Guide, Laboratory Equipment, Achievement Tests, 
and films. The textbook is written to sequentially develop, repeatedly 
using as much material as possible, a concept of the atomic model of 
matter, and it has laboratory experiments placed throughout to enhance 

? - — 

See Incomplete Data, page 23 



- 19 - 



the total development of the concept. The Teacher’s Guide is written 
to assist the teacher in conducting a course as it was intended and 
is very detailed in its descriptions and directions. The Laboratory 
equipment is especially designed for IPS and is packaged in lilt -form 
for easy distribution and use in schools. There are three series of 
IPS Achievement Tests: Series A, Series B, and Series C. Series A 

was used in this study. The films are designed to supplement the course 
materials, but they were: not systematically used in this ^t-udy because 
they are in an early stage development. 

The IPS course is designed to: 

(1) provide a foundation in subject matter and develop 

the appropriate attitudes of inquiry, coupled 
with the necessary experimental and mathematical 
skills, needed for non advanced study in science. 

(2) to serve as a terminal course for the student who will 

take few or no more science courses. 

The IPS course satisfied the criteria for a structured science 
course and can be considered hierarchical, as was required of the course 
to be used in this study. 

Procedure 

The seven classes, selected for this study, were chosen 
during the summer preceding the school year during which the IPS course 
was taught. The teachers were selected because they had taught IPS in 
the two preceding years, and they can be considered about equal in 
experience with IPS. A meeting was held during the summer to acquaint 
the teachers, and the principals of the schools in which they taught, 
with the purposes of the study and the mechanics of collecting the 
data. The items discussed at this meeting were: 

1. ) Purpose of the study. 

2. ) Administering of the tests. 

3 . ) Recording of the data . 

4. ) Grading. 

5 . ) Visitation by project director. 

6 . ) Dissemination of results. 

In examining the purpose of the study, care was taken to 
convince the teachers that no evaluation of their work was intended. 

The techniques being applied, they were told, were designed to measure 
the effectiveness of the course materials. They were told, however, 
that teacher differences could not be overlooked in the treatment of ^ata 
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The school counseling departments took charge of administer- 
ing the DAT tests, both the pretest and the posttest. While the pre- 
tests were not administered on the same day in each school, they were 
administered during the first week of school so that the measures of 
relevant basic abilities could be considered to be those with which the 
students began the course. The posttest of the DAT was ad m i ni stered 
during the last week of school. The project director provided what- 
ever assistance was needed for recording scores, provision of answer 
sheets for post testing, and posttest scoring service. 

The teachers administered all the IPS tests, as well as all. 
teacher-made tests, and recorded all data. The project director 
provided supervision, checks on the accuracy of data, and resource 
information about the course, the tests, and problems encountered in 
the conduct of the course. 

The teachers were told how to record the data on the data form-', 
and were told of the importance of each item to the total project. 

They were instructed to record data for transfer students separately 
and to record all the information available for each student. They 
were encouraged to get complete data on each student whenever possible, 
but that only data which was available should be reported. 

The age, sex, intelligence quotient, and reading score were 
to come from the cumulative records of the students. IQ was the 
score on the California Test of Mental Maturity, administered the 
previous year, and the reading score was the total score on the Iowa 
Silent Reading Test, also administered during the previous year. The 
DAT pretest scores were raw scores on the sub-tests of the Differential 
Aptitude Test, Form L, administered during the first week of the study. 
The IPS test scores were the raw scores on the IPS Achievement Test, 
Series A, administered during the execution of the course, and the 
grades were letter grades (A,B,C,D, or F) assigned by the teachers 
at nine -week intervals. The final grade was the letter grade assigned 
by the teacher. The DAT posttest scores were the raw scores on the 
post administration of the DAT tests. 

The teachers were told to assign grades in their usual manner, 
but were warned not to use the IPS Achievement Test scores in the same 
way that they used the scores from their own tests. The elements of 



\ copy of the data form can be found in Appendix F. The items were: 
age, sex, IQ, reading level, pretest DAT subtest scores, IPS Achieve- 
ment test scores, teacher's grades, and posttest DAT subtest scores. 



standardized testing were briefly explained to them. All teachers 
assigned grades on an A,B,C,D,F scale. 

Ifce teachers were told that frequent visits would be made by 
the project staff. The purpose of these visits were (l) to become 
acquainted with the school situation and the teachers, ( 2 ) to get a 
description of the classes which would help in classifying the data 
for analysis, (3) to observe problems that arose in connection with 
the IPS course, (U) to assist the teachers whenever they wished it, 
and (5) to answer questions about the project. Each class was visited 
at least three times during the study, and recorded observations were 
made in a survey form prepared for this purpose.^ 

The teachers and the administrators of Fairfax County accepted 
the study with interest and enthusiasm. They were eager to have some 
help in answering questions that they had asked themselves about the 
effectiveness of IPS. 



^A copy of the survey form as well as a description of the classes can 
be found in Appendix E of this report. 



RESULTS 



Incomplete Data 



Hie re were 931 students involved in the IPS course and included 
in this study. On each student., thirty- three items of information were 
recorded. One hundred forfcyy records bad to be deleted from the study 
because they contained too few items of information to be of value in 
the data analysis, the decision to exclude a record being made if 
there were missing more than four of the thirty-three items of inform- 
ation. Otoe hundred forty-eight of the remaining 791 records were also 
incomplete , but these records each had no more than four items missing 
with almost all of them having only one missing. Also, some items 
were missing with high frequencies. 

A count of the number of items missing and the number of points 
missing for each item was made. Uien, for each item having a 
sufficiently large number of missing points, ft regression equation was 
computed, 5 The regression equations were then used to fill in the 
missing points. 

Whenever a record was missing only one item, the point was 
f ille d with the score predicted on the basis of other information 
available from the record. For a record that had two missing items, 
the mean for one item was substituted to compute the second item. 

Then the computed item was substituted to recompute a value for the 
item for which the mean had been used. This procedure was used for 
items up to four in number. If a record had more than four missing 
items, it will be recalled, it was not used in the analysis. 

When there was a missing item for which no regression equation 
had been computed, the mean for that i+em was substituted. Oit of the 
lW3 incomplete records used, each having thirty-three items, the mean 
for a given item was substituted as a point only seven times. For all 
other substitutions, the prediction based on the regression equa+^on 
was used. 

While the regression equation varied in the worth of the 
predictions they produced, comparisons of records having similar 
scores on each i'cem revealed that the predicted scores were always 
more in line with the ’'expected 11 scores than would the means have been. 
Since no regression involved more than three means and most involved 



5See Appendix C- for regression equations. 



none, the 148 incomplete records used in the data analysis were, 
after completion, a valuable addition to the total data used in the 
analysis . 



Identification of Relevant Basic Abilities 



Relevant basic abilities are tho^e abilities with which the 
student must be equipped when he begins an instructional sequence — 
if be is to successfully cope with the sequence. According to the 
theory, correlations of measures of relevant basic abilities with 
measures of achievement at successive points in a hierarchical se- 
quence will be high. They will increase sharply for an ineffective 
course, they will remain about the same -for an effective course, and 
if they decrease the course is likely to be effective but inefficient. 
Correlations of measures of general abilities and abilities not rele- 
vant to the course, with achievement will be lower and they will be 
expected to remain somewhat stable (Gagne, 19^1) with the exception 
that irrelevant abilities may behave erratictlly. 

Inasmuch as the IPS course demands systematic observation 
and reasoning, careful correlation and proof of ideas, and extensive 
computation, Verbal Reasoning, Abstract Reasoning and Numerical Ability 
would be expected to be relevant basic abilities for the IPS course. 

If this is so, analysis of the actual correlations found in the study 
should confirm the fact. 

Clerical speed and accuracy having little or nothing to do with 
success in the IPS course, should behave as an irrelevant ability would 
be expected to behave. And, IQ being a measure of general ability, 
should behave so. If this is so, analysis of the actual correlations 
should confirm the fact and in so doing support the confirmation of 
the relevant basic abilities. 

Since other measured abilities cannot be categorized on the 
basis of what is known about the behavior of relevant basic, general, 
and irrelevant abilities, and what is known about the IPS course, the 
analysis of their correlations with achievement might be expected to 
indicate the appropriate category. 

Following is a 1st of the abilities measured in this study 
and their predicted behabior. 

1. Verbal Reasoning: relevant basic ability. 

2. Abstract Reasoning: relevant basic ability. 

3. Numerical Ability: relevant basic ability. 

k, IQ: general ability. 

5. Clerical Speed and Accuracy: irrelevant ability. 

6. Reading: uncertain. 

7. Space Relations: uncertain 

8. Language Usage (spelling): uncertain. 



9. Language Usage (sentences): uncertain. 

10. Mechanical Reasoning: uncertain. 

The data were divided into twenty-eight groups for this 
analysis. Classifications, based on the success of the regression 
models in predicting missing points, was by sex, ability (measured 
by IQ), and school. There were two categories each for sex and ability, 
the dividing point for ability being the mean IQ, and seven categories 
of school. Correlation coefficients were computed for each of the ten 
measures of ability with scores on Achievement Test I, Achievement 
Test H, and Achievement Test III (when available). The patterns of 
correlations were examined to determine whether the abilities expected 
to be relevant basic behaved as relevant basic abilities were expected 
to behave and to see if additional measured abilities might be called 
relevant basic. The patterns were also examined to see how expected 
general and irrelevant abilities behaved. 

In twenty-six of the twenty-eight cases, the correlations of 
clerical speed and accuracy with Achievement Tests I, II, and HI were 
low in magnitude. One of the two exceptions was a group of low IQ 
girls in school number seven which had only five students. This can 
be discounted because of the small number. The other was a group of 
high IQ girls in school number 1 which had nineteen students, but the 
high correlation for this group was only .55* All other correlations 
were considerably lower, most being nearly zero in magnitude. This 
was in accordance with predictions based on theory. 

Only two other measured abilities exhibited similar patterns 
in every group. These were Language Usage (spelling) and Language 
Usage (sentences). Only occasionally did these correlations excede 
•50, and they were usually considerably lower. Of the correlation 
coefficients resulting from correlations of Language Usage with 
achievement, the highest were found in two groups of high ability boys 
and high ability girls from the same school when the correlations involved 
Language Usage (sentences). 

On the basis of these results it was concluded that Clerical 
Speed and Accuracy as expected, was an irrelevant ability for the IPS 
course and that Language Usage (spelling) was also an irrelevant ability. 
Language Usage (sentences) was probably an irrelevant ability, but the 
evidence was not conclusive. 

The correlation coefficients for these three abilities, using 
only high ability students appear in Table 1. 
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TABLE 1 



CORRELATION COEFFICIENTS FOR SCORES 
ON ABILITIES NOT 
RELEVANT TO THE IPS COURSE 
WITH SCORES ON ACHIEVEMENT 
FOR HIGH ABILITY STUDENTS 







Ach. Tst. 


Ach. Tst. 


Ach. Tst 






I 


II 


III 


High IQMl 8 


CSA D 


.42 


.43 


- 


(iQ=125;n=32) 


LUspC 


.57 


.65 


- 




LUse d 


.65 


.68 


• 


High KF1 


CSA 


.31 


.55 


- 


(Dj:i2Tjn»19) 


LUsp 


.64 


.60 


- 




LTJse 


.80 


•75 




High IQM2 


CSA 


.20 


.12 


- 


(K= 126 jn= 38 ) 


LUsp 


.20 


.30 


- 




LUse 


.05 


- .05 




Hirti IQF2 


CSA 


.43 


.18 


- 


(lQ*12U;n=37 ) 


LUsp 


.52 


.28 


- 




LUse 


• 75 


.51 




High IPltt 


CSA 


- .35 


- .04 


— 


(f§-124;n=23) 


LUsp 


.24 


.25 


- 




LUse 


.43 


.50 


* 


High IQF3 


CSA 


.23 


.25 




(fS»124;n»4o) 


LUsp 


.43 


.29 


- 




LUse 


.66 


.55 


— * 


High IQM4 


CSA 


.20 


- .04 


- 


'lQ=124;na32) 


LUsp 


.30 


.19 


- 




LUse 


• 37 


.20 


- 
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TABLE I CONTINUED 







Ach. Tst. 


Ach. Tst. 


Ach. Tst 






I 


II 


III 


High IQF4 


CSA 


.15 


.18 


, m — 


(ift»126;n»37) 


LUsp 


.40 


.08 


- 




LUse 


.40 


.22 


— 


High IQM5 


CSA 


- .08 


- .14 




(IQ-I25;n»20) 


LUsp 


.22 


.15 


- 




LUse 


.20 


.25 




High IQF5 


CSA 


- .12 


- .08 




(I§- 129 ;n-l 8 ) 


LUsp 


• 50 


.32 


- 




LUse 


.42 


• 37 




High IQM 6 


CSA 


.08 


.04 




(rQ- 12 ^;n= 26 ) 


LUsp 


.41 


.24 


- 




LUse 


.33 


.26 


• 


High IQF 6 


CSA 


.25 


- .08 




(I§»124;n»20) 


LSUsp 


.16 


- .08 ! 


- 




LUse 


.28 


.23 




High IQKT 


CSA 


.13 


- .09 


.08 


(El.l32;n-25) 


LUsp 


• 37 


.18 


.41 




LUse 


• 35 


• 59 


.69 


High IQF7 


CSA 


- .12 


- *07 


.00 


(lQ-125;n»42) 


LUsp 


- .04 


.15 


.08 




LUse 


• 53 


.38 


•19 



1 ERIC 



TABLE I CONTINUED 



a High IQ Males from school number 1. 
^Clerical Speed and Accuracy 
c Language Usage (spelling) 

^Language Usage (sentences) 



The correlation coefficients for the irrelevant abilities using 
only data from the low ability students appear in Table 2. 

TABLE 2 

CORREIATION COEFFICIENTS FOR SCORES 
ON ABILITIES NOT 
RELEVANT TO THE IPS COURSE 
AND SCORES ON ACHIEVEMENT 
FOR LOW ABILITY STUDENTS 







Ach. Tst. 


Ach. Tst. 


Ach . Tst 






I 


II 


III 


Low IQ tfL a 


CSA b 


•37 


- .18 




(?Q*99jn*26) 


LUspC 


.23 


•05 


- 




Luse^ 


• 30 


.18 




Low IQF1 


CSA 


.10 


.24 




(lQ-100;n-32) 


LUsp 


• 32 


.24 


- 




LUse 


• 37 


.*9 




Low IQM2 


CSA 


• 31 


.40 




(lQ-100;n-27) 


LUsp 


.06 


.05 


- 




LUse 


• 37 


.16 




Low IQF2 


CSA 


.11 


- .18 


mm 


(j&.100;n-33) 


LUsp 


• 30 


.12 


- 




LUse 


.22 


.34 


- 


Low IQH3 


CSA 


.14 


.10 




(TO-100jn»33) 


Lusp 


- .10 


.02 


- 




LUse 


• 43 


• 37 




Low XQF3 


CSA 


.40 


.22 




(^.97;n.38) 


LUsp 


.27 


.22 


- 




LUse 


.2k 


• 50 
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TABLE 2 CONTINUED 







Ach. Tst. 


Ach. Tst. 


Ach. Tst 






I 


II 


III 


Low IQM4 


CSA 


• 3* 


.04 


_ 


(fQ»96;na3l) 


LUsp 


- .12 


.03 


- 




LUse 


.13 


.25 




Low IQF4 


CSA 


.27 


.06 




(lQ-9^;n-29) 

» 


LUsp 


• l 6 


- .14 


- 


LUse 


.24 


.00 




Low IQM5 


CSA 


.30 


.13 


mm 


(ft. 9 i 5 n. 33 ) 


LUsp 


.41 


.28 


- 




LUse 


c^\ 
• S'* 


& 


' 


Low IQF5 


CSA 


.11 


.17 




(lQ«89;n»29) 


LUsp 


.35 


.19 






LUse 


M 


.28 




Low IQM6 


CSA 


.01 


„24 




(IQ-I00;n«29) 


LUsp 


- .18 


- .08 


- 




LUse 


.29 


.44. 




Low IQF6 


CSA 


.16 


- -JT 




(iQ«101;n«34) 


LUsp 


.03 


.±3 


- 




LUse 


.24 


.12 




Low IQM7 6 








_ 


(IQ-I08;n»3 ) 


- 


- 


- 


- 


Low IQFT 6 


_ 


wm 


- 


- 


(S*110;na5) 


- 


- 


- 






mm 
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TABLE 2 CONTINUED 



a High IQ Males from school number 1. 

^Clerical Speed and Accuracy 
c Language Usage (spelling) 

^Language Usage (sentences) 

e The numbers in these groups are so small that the correlation 
coefficients are not meaningful. 
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No measured ability behaved clearly in every instance £8 a 
general ability would be expected to behave. Patterns of correlation 
coefficients of measures of IQ with measures of achievement exhibited 
wide variation 8 when classified by sex, by school, or by ability level. 
The widest variation was found among girls of high ability where corr- 
elation coefficients ranged from near zero to about .80. Figure 1 
illustrates this wide variation. The patterns were scattered, although 
not over as wide a range for low ability girls. The patterns of 
correlations of measures of IQ with achievement for boys were clustered 
most and behaved more nearly as they would be expected to behave if IQ 
were a genei^l ability. In the case of boys of low ability, the 
patterns were very nearly what would be expected of a general ability. 
Patte rns for liigh ability boys and patterns for low ability boys appear 
in Figures 2 and 3> respectively. The correlations were higgler among 
the high ability boys than they were among the low ability boys. This 
difference in the magnitudes of the correlation coefficients was not 
evidenced among the girls * 

IQ, then, was foun: to behave as a general ability would be 
expected to behave when patterns cf correlation coefficients for low 
ability boys were examined. While the behavior of IQ when all boys were 
considered was not entirely clear, the patterns e x a min ed were not such 
that the assumption that IQ was a general ability in the case of boys 
could be easily refuted. This was not so with girls. Either there was 
a source of considerable variance which was uncontrolled In the data 
analysis, or IQ was not a general ability for the IPS course when only 
girls were considered. 

When the patterns of correlations of measures on reading 
ability with measures of achievement were examined it was found that 
reading behaved more as a general ability would be expected to behave 
than did IQ. The patterns for reading ability using low ability girls 
appear in Figure k. This is the pattern that was most nearly what was 
expected of a general ability on the basis of the theory. It is 
included here for this and for an additional reason, the behavior of 
coefficients of correlation for school number k to which attention will 
be called in the discussion on measures of effectiveness. 

Although the patterns of coefficients of correlation of measures 
on reading ability with measures on achievement were somewhat variable, 
though less variable than those for IQ, the data analysis indicated that 
reading ability was a general ability. The differences by sex found in 
the behavior of patterns of correlations when IQ was e xamine d were not 
in evidence in the patterns exhibited when rea din g ability was examined. 

Of the other abilities measured, only Space Relations ability 
e xhib ited, patterns of correlation which were indicative of a general 
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ability. Although there were small differences by ability level, 
patterns of correlation coefficients of measures on Space Relations 
ability with measures on achievement behaved more consistently as a 
general ability would be expected to behave than any other u^easured 
ability. Patterns found for high ability boys and for low ability 
boys appear in Figures 5 and 6, respectively. 

Hie behavior of correlations of Space Relations ability with 
achievement indicated that it was a general ability for the IPS 
course. Since there were no large differences in the patterns for 
boys and those for girls, this ability was a general ability for both. 

Of the measured abilities expected to behave as relevant 
basic abilities none did so consistently. Abstract Reasoning 
ability exhibited no discerhable pattern and could not, on the basis 
of the data analysis, be called a relevant ability, numerical Ability 
and Verbal Reasoning Ability, however, exhibited the expected behavior 
but only when the data were grouped by sex, and then the expected 
behavior was found only among boys. 

Hie best example of the behavior of an ability expected to 
be relevant basic was found among boys of high ability when patterns 
of correlations of measures on Verbal Reasoning ability with measures 
on achievement were examined. Haeae patterns appear in Figure 7* 

From Figure 7 it is easily determined that the correlations are high, 
that they cluster about the same point for Achievement Test I, and 
that they branch from that point for Achievement Test H. 

It is this kind of branching that can be useful in measuring 
the effectiveness of instructional sequences and in making comparisons 
of the effectiveness of a sequence in different instructional settings. 

While in no other instances were the patterns of correlations 
as clearly what was expected of relevant basic abilities as they were 
in this group of high ability boys where Verbal Reasoning Ability was 
correlated with achievement. Verbal Reasoning ability and Numerical 
Ability exhibited patterns which indicated strongly that they were 
relevant basic abilities for the IPS course when only boys were con- 
sidered. Hie se abilities were not found to be relevant basic for girls. 

The data analysis revealed no patterns of correlations of 
Mechanical Reasoning ability with achievement from whibh it could be 
decided whether Mechanical Reasoning ability was irrelevant, general, 
or relevant basic. 



In summary, the data analysis revealed that Clerical Speed 
and Accuracy, which was expected to be an irrelevant ability for the 
IPS course, exhibited the expected patterns of correlation with 
achievement. Language Usage (spelling) also exhibited the patterns 
expected for an irrelevant ability, and Language Usage (sentences) 
exhibited s imil ar patterns. It was concluded, then, that Clerical 
Speed and Accuracy was in fact an irrelevant ability and that 
Language Usage (spelling) was also. The analysis did not clearly 
indicate whether Language Usage (sentences) was an ability not 
relevant to the IPS course. 

IQ, expected to behave as a general ability, was found to 
behave as expected only for boys and even then not too nearly. The 
expected patterns were most nearly represented in boys of low ability. 
Patterns for girls were extremely erratic. 

Patterns of correlations of reading ability with achievement 
were more in accordance with the behavior predicted for general 
abilities than were those of IQ. The patterns most nearly like those 
expected were found among low ability girls. Patterns of correlations 
of Space Relations ability with achievement were like those expected 
of a general ability. It was concluded, then, that reading ability and 
Space Relations ability were general abilities for the IPS course and 
that IQ was a general ability for the course only for boys. 

Cf the three measured abilities expected to behave as 
relevant basic abilities, only Numerical Ability and Verbal Reason- 
ing ability did so with any regularity. The data clearly revealed 
that ^Abstract Reasoning ability was not an ability relevant to the 
IPS course. Patterns of correlation of Verbal Reasoning ability 
with achievement were most nearly in agreement with predictions 
in the case of hi ^3 ability boys. Patterns of correlations of 
Verbal Reasoning ability and Numerical Ability with achievement 
were generally what was expected of relevant basic abilities but 
only for the boys. Patterns among girls were quite variable. It 
was concluded, then, that Verbal Reasoning ability and Numerical 
Ability were relevant basic abilities for the IPS course only when 
boys were considered. 

Mechanical Reasoning ability, the remaining measured 
ability, exhibited no clearly discemable patterns of correlations 
with achievement. Its status remained uncertain. 




- 35 - 



correlation coefficient 



1.0 





i 3t m. 



Achievement Test 

Fig. 1. - Patterns of coefficients of correlation of 
IQ with achievement for high ability girls. Data In TKble 3. 
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TABLE 3 



COEFFICIENTS OF CORRELATION OF IQ 
WITH ACHIEVEMENT FOR HIGH AND 
LCW ABILITY GIRLS 



School 


Ability Level (h) 


Ach.Test 

I 


Ach.Test 

II 


Ach.Test 

III 


1 


High (19) 
Low 


■77 

.46 


•79 

.46 


- 


2 


High 


•70 


.67 


- 




Low 


- .10 


• 39 






High 


.46 


.41 


- 


J 


Low 


.24 


.44- 




4 


High 


.32 


.23 


- 




Low 


.64- 


.00 




5 


High 


.65 


.80 


- 




Low 


• 39 


.29 




6 


High 


.09 


- .01 






Low 


*23 


.36 




7 


High 


•15 


.23 


.14 


1 


Low 


.15 


.50 


• 51 
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Fig. 2. - Patterns of coefficients of correlation of 
IQ with achievement for high ability boyB. Data in Table 
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Achievement Tests 



Fig. 3* - Patterns of coefficients of 
with achievement for low ability boys. Data 



correlation of 
in Table 4. 
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TABLE 4 



COEFFICIENTS OF CORRELATION OF IQ 
WITH ACHIEVEMENT FOR HIGH AND 
LOW ABILITY BOYS 



School 


Ability Level (b) 


Ach. Test 


Ach. Test 


Ach. Test 






I 


II 


III 




High p2) 


.69 


.64 






Low (26) 


.32 


.32 


* 


2 


High (38) 


.4o 


.47 


m 




Low (27) 


.46 


.52 


* 


3 


High (23) 


• 53 


•34 






Low (33) 


.47 


.50 




4 


High (32) 


.52 


.ko 






Low (31) 


.25 


•H 


mm 


5 


High (20) 


• 34 


.46 






Low (33) 


.44 


.21 


m 


6 


High (26) 


.58 


.52 






Low (29) 


.50 


.33 


** 


7 


High (25) 


.56 


.40 


.63 




Low { 3) 


.23 a 


•97® 


- .27 s 



coefficient of correlation 



1.0 



0.8 •’ 



0*(i 




Achievement Test 



Fig. 4. - Patterns of coefficients of correlation of 
reading ability vitii achievement for low ability girls. Data 
in Table 5* 
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TABLE 5 



COEFFICIENTS OF CORRELATION OF 
READING ABILITY WITH ACHIEVEMENT 
FOR LOW ABILITY GIRLS 



School 


(n) 


Ach. Test 
I 


Ach . Test 
IX 


Ach. Test 
III 


1 


(32) 


• 37 


.36 


- 


2 


(33) 


.46 


.50 


- 


3 


(38) 


.41 


.41 


- 


4 


(29) 


.40 


- .16 


- 


5 


(29) 


.50 


.30 


- 


6 


(3*0 


.25 


.42 


- 


7 


( 5) 


- -37 a 


- .47 s 


- .39* 



a Based on n = s; not plotted in Figure 4 



coefficient of correlation 




^>-1 t- i 



i 



TL 
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Fig. 5- - Patterns of coefficients of correlation of 
Space Relations ability with achievement for high ability boys. 
Data In Table 6. 
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a ar m 

Achievement 

Fig. 6. - Patterns of coefficients of correlation of 
Space Relations ability with achievement for low ability boys. 
Data in Table 6, 
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TABLE 6 



COEFFICIENTS OF CORRECTION OF 
SPACE RELATIONS ABILITY WITH 
ACHIEVEMENT FOR HIGH AND 
LOW ABILITY BOYS 



School 


Ability Level (n) 


'cL, 'Test 


Ach. Test 


Ach. Test 








II 


III 


1 


Higji (32) 


.58 


.40 1 


1 . 




Low (26) 


.36 


^18 




2 


High (38) 


.64 


.62 


— 




Low (27) 


.48 


.25 




3 


Higil (23) 


.36 


.54 






Low (33) 


.64 


• 59 




4 


High (32) 


.44 


• 58 






Low (31) 


.17 


.25 i 




5 


High (20) 


.38 


.46 






Low (33) 


• 53 


•35 


* 


6 


High (26) 
Low (29) 


.45 

.48 


.36 

.49 


- 


7 

4 


High (25) 


.58 


.47 




Low ( 3) 


•76 a 


.63 a 


- .78 a 






1 



a Based on n • 3; not plotted 



Table 8 is a summary of the changes in the predicted results 



as a consequence 



of data analysis. 
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Fig. 7 - Patterns of coefficients of correlations of 
Verbal Reasoning ability with achievement for higi ability 
boyB. Data in Table 7* 
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TABLE 7 



COEFFICIENTS OF CORRELATION OF 
VERBAL REASONING ABILITY WITH 
ACHIEVEMENT FOR HIGH AND 
LOW ABILITY BOYS 



School Ability Level (n) Ach. Test Ach. Test Ach. Test 

I II III 



1 


High 


(32) 


•59 


•77 






Low 


(26) 


•30 


•5 6 


* 


2 


High (38) 


52 


•5’+ 


«• 




Low 


(27) 


.19 


.1*1+ 


* 


3 


High 


(23) 


•59 


Mi 






Low 


(33) 


.6k 


.50 


' 


1* 


High 


(32) 


.58 




— 




Low 


(3D 


.1*2 


.36 


* 


5 


High 


(20) 


.56 


•71 


• 




Low 


(33) 


.5^ 


• 53 




6 


High 


(26) 


.56 


•5^ 


• 




Low 


(29) 


.63 


.50 


' 


7 


High 


(25) 


.61 


.1*2 


*71 




Low 


( 3) 


l.Otf* 


- .03 a 


-1.00* 

1 



a Based on n = 3; not plotted 
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Jig. 8 - Patterns of coefficients of correlation of 
Verbal Reasoning ability with achievement for low ability boys. 
Data in Table 1. 








TABLE 8 



SUMMARY OF THE TEST OF PREDICTED 
BEHAVIORS OF MEASURED 
ABILITIES 



ABILITY 


PREDICTED 

BEHAVIOR 


ACTUAL 

BEHAVIOR 


1 . Verbal Reasoning 


relevant basic 


relevant basic (boys only) 


2. Numerical Ability 


relevant basic 


relevant basic (boys only) 


3 • Abstract Reasoning 


relevant basic 


uncertain 


IQ 


general 


general (boys only) 


5* Clerical Speed and 
Accuracy 


irrelevant 


irrelevant 


6 . Reading 


uncertain 


general 


7 • Space Relations 


uncertain 


general 


8 . Language Usage 
(spelling) 


uncertain 


irrelevant 


9* Language Usage 
(sentences) 


uncertain 


uncertain 


10. Mechanical 
Reasoning 


uncertain 


uncertain 
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Measures of Effectiveness 



According tc tne theory, a set of rapidly increasing 
coefficients of correlation of a relevant basic ability with achieve- 
ment in a structured program will reflect "drop out" among students 
of low basic ability and will be indicative of an ineffective course. 

A set of rapidly decreasing coefficients of correlation will reflect 
"over achievement" and w ill be indicative of an effective but 
inefficient course. A set of high correlations of close to zero 
slope will indicate an effective course. 

After the relevant basic abilities for the IPS course had 
been identified, predictions of achievement based on the theory 
were made in cases where the abilities were behaving as the theory 
predicted they should. These predictions were then checked against 
what actually happened in the classes to provide a check on the 
ability of the theory to detect effectively taught courses. 

Verbal Reasoning ability was found to be a relevant basic 
ability for boys, and its behavior most nearly conformed to theoretical 
predictions for boys of high ability. The relative achievement of 
classes of high ability boys, then, can easily be predicted using 
Figure 7. Since the lines drawn for schools numbered 1 and 5 
increase sharply from Test I to Test II, a large drop in the 
achievement on Test II relative to Test I can be predicted for both 
these groups of boys. The lines drawn for schools numbered 2 and 6 
have near zero slope. It can be predicted that there will be little 
or no difference in the achievement on Test II relative to that on 
Test I for these two groups. Since the lines drawn for schools 
numbered 3> 4 and 7 drop, it cannot be predicted with assurance what 
will happen. Such a drop is indicative of "over achievement" and 
one would expect the achievement on Test II relative to Test I to 
be higher. This would certainly be expected to be the case for 
school number 4, in which it was known from visits to the classes 
in that school that the teacher coached the students for the IPS 
achievement tests. This effect was dramatic in the case of low 
ability girls in that 3 school. Attention was called to the marked 
behavior of correlations among low ability girls when Figure 4 was 
discussed. But this differential could go either way, depending on 
which test was better coached. From the direction of line number 4 
in Figure 7, an increase in achievement on Test II relative to Test I 
would be expected. The marked drop in correlations from Test I to 
Ttest II in school 4 was found in every patxern in which low ability 
girls were studied. The effect was evident, although not always as 
pronounced, in groups of high ability girls. Large changes were 
evidenced in groups of boys, but there was not always a sharp drop. 
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A summary of the predictions based in the theory appears on 
Table 9 . The relevant basic ability used was Verbal Reasoning 
ability and the comparisons were made for high ability boys. 
Comparisons were made by converting the deviations of the group 
means from the total mean for all groups on the given test to 
standard units. 

The expected agreement of findings with predictions based on 
theory was found. There was some doubt about what happened to 
achievement when the pattern of correlations showed a drop, but 
little doubt about what achievement did when the pattern showed an 
increase or remained stable. Apparently, correlations can show a 
decrease while achievement remains relatively stable but an increase 
in correlations indicates a drop in achievement. 

The theory would predict, as is evident from Figure 7> that 
the performance of group 7 on Test III would drop relative to the 
other two tests. A 2 of .37 calculated using means and standard 
deviations from the standardizing data for the IPS tests, confirmed 
this prediction. 
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TABLE 9 



SUMMARY OF COMPARISON OF PREDICTED CHANGES 
IN ACHIEVEMENT WITH ACTUAL 
CHANGES FOR HIGH ABILITY 
BOYS USING VERBAL REASONING 
ABILITY AS RELEVANT BASIC 



Group 


Mean 
Test I 


Mean 
Test II 


Z 

I 


Z 

II 


Predicted 

Change 


Actual 

Change 

ZI - Z2 


1 


14.13 


15.22 


.51 


.20 


Large drop 


- .31 


2 


13.74 


12.08 


.42 


•53. 


no change 


.11 


3 


16.22 


18.87 


•97 


.86 


increase 


- .11 


4 


13.94 


17.22 


• 47 


•38 


increase 


- .09 


5 


17-40 


19.30 


1.23 


.94 


large drop 


- .29 


6 


12.88 


14.96 


0.24 


• 15 


no change 


- .09 


7 


17.O8 


j 19.24 


1.16 


• 93 


increase 


- .23 
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A s umma ry of predieitions based on the theory for the low 
ability boys appears in Table 10. Verbal Reasoning ability was 
again used as the relevant basic ability. The predictions were 
derived from the patterns of correlations in Fi^re 8. The t e y 
does not hold up quite as well for this group, but it should be noted 
that the patterns of correlation coefficients in tms ®™ U P^ S “ 
close to that required by the theory as was the <»se with the group 
of high ability boys. It is of interest to note that the coaching 
?n Tool number /is in this case revealed in the expected increase 
in achievement on Test II relative to Test I. This, coupled witt, , 
the sharply decreasing patterns among low ability aMlity 

the effect is more difficult to detect among students of high ability. 



Patterns of correlation coefficients of Numerical Ability with 
achievement for high ability boys and for low ability boys appear in 
Figures 9 and 10, respectively. If Numerical Ability is considered 
relevant basic and these patterns are used to predict changes in 
achievement, agreement with the predictions made using Verbal Reason- 
ing ability as relevant basic, while not perfect, is nevertheless good. 
Predictions, or measures, of effectiveness, which be essentially the 
same in the two cases. The most notable exceptions are predictions 
for high ability boys in schools 5 and 6. If the variability of the 
initial correlations, those of the relevant basic ability with 
achievement Test I, could be carefully controlled, there is strong 
evidence that measures of effectiveness based on the theory and 
using either relevant basic ability could be corroborated almost 
completely by using the other. 

In summary, both the correlation patterns and the changes in 
achievement between Test I and Test II indicated that school number 
2 and, probably school number 6 were teaching an effective course for 
hidi ability boys. The patterns of correlations indicated that schools 
3, 4, and 7 were teaching an effective but inefficient IPS course for 
boys* of high ability. The changes in achievement between Test I and 
Test II however, indicated that school number 7 was teaching an in- 
effective course. Both the correlation patterns and the changes in 
achievement indicated that schools number 1 and 5 were teaching an 
ineffective IPS course for high ability boys. School number 4, which 
had been suspect because it was known that the teacher in that school 
"taught for" the IPS achievement tests, had patterns of correlations 
which confirmed the suspicion. 

The correlation patterns and the changes in achievement 
indicated that school number 5 was teaching an effective IPS course 
for low ability boys. The correlation patterns for schools 3 and o 
indicated that they were teaching an effective but inefficient course,. 
Changes in achievement did not confirm this in both cases. The 
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correlation: patterns indicated that schools 1 and 2 were teaching 
an ineffective IPS course for low ability boys, but changes in 
achievement failed to confirm this in both cases. Suspicions aooun 
school k were again confirmed by the correlation patterns. In the 
case of low ability boys, the changes in achievement between Test 1 
and Test II strongly supported the suspicious correlation patterns. 
There were not enough students in the low ability group in school 7 
to make a judgment about the effectiveness of the IPS course 
advisable in that situation. 

A summary of the determination of the effectiveness of the 
IPS course is given in Table 12. 

These results were in good agreement with the results of 
observations made on visits to the IPS classes.* On the basis of 
these observations, one would have expected the teacher in school 2 
to be effective with both high and low ability students. It would 
also be expected that the teacher in school 3 would have been 
effective with at least one group. 



*See Appendix E 



o 
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TABLE 10 

SUMMARY OF COMPARISON OF PREDICTED CHANGES 
IN ACHIEVEMENT WITH ACTUAL 
CHANGES FOR LOW ABILITY 
BOYS USING VERBAL REASONING 
ABILITY AS RELEVAiJT BASIC 



Group 


Mean 
Test I 


Mean 
Test II 


Z 

I 


z 

II 


Predicted 

Change 


Actual 

Change 


1 


9-35 


10.42 


- . 5 ^ 


- .68 


Large drop 


- .14 


2 


8.63 


10.59 


- .69 


- .65 


Large drop 


4. .04 


3 


10. W 


12.30 


- .29 


- .34 


Increases 


- .05 


4 


11.35 


15.10 


- .10 


f -IT 


No change 


4 .27 


5 


7-76 


10.00 


- .91 


- .76 


No change 


+ -15 


6 


9.76 


10.62 


- .45 


- .65 


Increase 


- .20 
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Fi^. 9 - Patterns of coefficients of correlation 
of Numerical Ability with achievement for higi ability boys. 
Data in Table 11. 
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Fig. 10 - Patterns of coefficients of correlation 
of Ntmerical Ability vith achievement for Low ability boys. 
Data In Table 11. 





TABLE 11 



COEFFICIENTS OF CORREIATION OF 
NUMERICAL ABILITY WITH 
ACHIEVEMENT FOR HIGH AND 
LCV ABILITY BOYS 



School 


Ability Level (n) 


Acb. Test 


Ach. Test 


Ach. Ttest 






I 


II 


III 


1 


r 

High (32) 


.32 


• 50 


• 




Low (2 6 ) 


.32 


• 45 




2 


Higi (38) 


.52 


.60 


- 




Low <;(27) 


• 39 


.46 


* 


o 


High (23) 


.56 


.60 






Lew (33) 


.48 


.38 


** 


4 


Hi#> (32) 


.48 


.53 


• 




Low (3l) 


.29 


.53 




5 


High (20) 


• 53 


• k 3 


- 




Low (33) 


.58 


.47 




6 


High (26) 


• 55 


; 

\ -71 


- 




Low ( 29 ) 


.25 


.11 




7 


High (25) 


.69 




.76 




Low ( 3) 


.07 a 


•99 


- .11 



a Based on n * 3; not plotted 
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TABLE 12 

SUMMARY OF MEASURES OF THE 
EFFECTIVENESS OF THE IPS 
COURSE FOR HIGH AND 
LOW ABILITY BOYS 



Effectiveness 



School (Ability) Effective Effective but Ineffective 

Inefficient 



1 


High 


. 




1,8,3 




Low 




- 


l,z,3 


2 


High 


1,2,3 








Low 




— 


1,2 


3 


High 




1,2 






Low 


• 


1,2 


- 


k 


Higi 




1,2,3 






Low 


1 




2 


5 


High 






1,2,3 




Low 


1 


2 




6 


High 


1,3 








Low 


• 


1,2 


4 » 


7 


High 


. 


1 1,2 






Low 


not determined 

1 





Code: 1 



2 



3 



confirmed using Verbal Reasoning as a relevant basic ability 
confirmed using Numerical Ability as a relevant basic ability 
confirmed using Achievement changes 
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CONCLUSIONS ,uMD RECOMMENDATIONS 



Conclusions 



The study reported here had as its primary objective the 
testing of a prototype model for determining the effectiveness of 
structed instructional materials. The test of the model was to 
be a severe one. The model was required to hold up in a real 
school situation and for a course in introductory physical science (IPS)* 
If the prototype model held up under this test, a mathematical 
expression was to be derived which could be used to add an element 
of quantity to measures of the effectiveness of such instructional 
materials . 

The prototype model was based on a theoretical foundation. 

The theory predictions of the effectiveness of structured instructional 
materials based on patterns of correlation coefficients of measures 
of basic abilities relevant tb the materials with measures of 
achievement at successive points in the structured sequence of 
materials . There were several questions that had to be satisfactorily 
answered, however, before the model could be put to a test. These 
questions were; 

1. What measured abilities were relevant to the IPS course? 

2. Whht patterns of correlation coefficients indicate an 

effective course? 

3. What unit should be used to divide the course when the 

coefficients are plotted to give a pattern? 

4. How can these patterns be mathematically described 

so that they will provide a quantitative measure 

of effectiveness? 

The study was successful in dealing wit. the first two 
questions. Patterns predicted on the basis of theory were found 
among the patterns plotted from the data, and there was good agree- 
ment between the expected behavior of measured abilities of a 
specified type and the behavior these abilities actually exhibited. 
Agreement of predicted patterns and actual patterns, however, was 
in most instances found to be a function of ability (as measured by 
IQ tests) and of sex. 

Abilities which were expected to be irrelevant to tbe IPS 
course were found to exhibit the predicted patterns independent of 
ability and of sex. Abilities expected to be general with regard to 
the IPS course were found to exhibit mixed ^ttems. IQ was found 




- 61 - 



to behave as a general ability for boys only while reading ability 
and Space Relations ability were found to behave as general 
abilities for all students. Abilities expected to be relevant and 
basic to the IPS course were found to exhibit the predicted patterns 
only for boys, the patterns most nearly what were expected being 
found among high ability boys. 

In the cases where an ability relevant to the IPS course 
could be identified the patterns of correlation coefficients were 
used to make predictions of the effectiveness of the IPS course for 
the groups of students for which the relevant basic ability could be 
identified. When tnese predictions were compared with changes in 
achievement which would be expected for effective and ineffective 
courses, the results were very good. 

On the basis of the theory and the data analysis of the study 
the following things were concluded: 

1. Relevant basic abilities as measured for the IPS course 

were not independent of the level of general 

intelligence. 

2. Relevant basic abilities as measured for the IPS course 

were not independent of sex. 

3. Irrelevant abilities as measured for the IPS course were 

independent of the level of general intelligence as 

well as of sex. 

4. The concept of genera J. ability as measured for the IPS 

course was open to serious doubt. 

% The model used for measuring the effectiveness of the 

IPS course retained much of its original promise. 

6. Verbal Reasoning ability and Numerical Ability were 

relevant basic abilities for the IPS course. 

7. The IPS course was effective in some but not all 

situations . 

Measures of the effectiveness of the IPS course, and thus 
the test of the prototype model, were severely limited because of 
the failure of any class to complete the IPS course and to take all 
four achievement tests. Two points in the correlation patterns is 
far too small a number to make determinations of effectiveness by 
the technique used in this study and sure procedure. Yet it was 
found that, as far as the data could test it, the model was a sound 
one. 



With regard to questions 3 and 4, the fact that no class 
completed the IPS course and took all four tests precluded the 
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possibility of attacking the problem of quantifying measures of 
achievement. only relationship possible when only two points 

are available is a line or one. Although a detailed analysis of 
the IPS textbook was conducted, dividing the course into units for 
the data analysis was not attempted due to the limited number of 
IPS achievement test scores available for the data analyses. 

In addition to the test of a prototype model for determining 
effectiveness, it was hoped at the outset of this study that a model 
for dete rmining whether the IPS course could be considered structured 
could also be subjected to a test. The test of this model depended 
upon the studen'ts completing .the IPS course and a measure of their 
final achievement. Since no class finished the course, the model 
could^nfctfcetteSfced. 



Recommendations 

On the basis of the experiences of this study and the results 
obtained from it, several recommendations can be made: 

1. The study revealed that patterns of correlations which 

showed a sharply increasing trend were indeed useful 
in spotting situations in which the IPS course was 
ineffective. Further study is needed on what is 
indicated by patterns which show a sharp decrease. 

As was indicated by the patterns for a situation where 
coaching was known to be going on, sharply decreasing 
patterns may be indicative of the best or most 
effective course, assuming, of course, that the 
tests of achievement have not been in llidated 
because of coaching. 

2. erratic behavior of general abilities measured in 
this study suggests that tie concept of general 
ability and its theoretically predicted behavior should 
be given further attention. The fabt that both measures 
of IQ and of reading ability in this study were composite 
scores might suggest that c orre.le.ti ons of general 
abilities from point to point ir. a structured sequence 
mi git remain stable because tho effects of more specific 
abilities are masked when they are combined into a 
composit score. This idea was supported in this study 
by the behavior of correlations involving Verbal 
Reasoning ability, but it was not supported by the behavior 
of correlations involving Space Relations ability, which 
itself behaved as a general ability would be expected to 
behave. The concept of general ability needs further study. 
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3. Future studies using tne models tested in this 

study should he done in a carefully controlled 
"laboratory” setting using programmed or 
program-like instructional sequences. Division, 
of the instructional sequences into the appropriate 
units probably should be accomplished by dete rmin ing 
average equal times for completing blocks of the 
instructional sequence. In addition to the carefully 
controlled environmental setting, the important 
variables, ability and sex, should be controlled. 

4. Some method of determining relevant basic abilities for the 

IPS course when girls take it should be sought. Ibis 
problem of differences by sex should be studied further. 
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